Feature extraction and matching for biometric authentication

ABSTRACT

In a feature extraction and pattern matching system, image sharpening can enable vascular point detection (VPD) for detecting points of interest from visible vasculature of the eye. Pattern Histograms of Extended Multi-Radii Local Binary Patterns and/or Pattern Histograms of Extended Multi-Radii Center Symmetric Local Binary Patterns can provide description of portions of images surrounding a point of interest, and enrollment and verification templates can be generated using points detected via VPD and the corresponding descriptors. Inlier point pairs can be selected from the enrollment and verification templates, and a first match score indicating similarity of the two templates can be computed based on the number of inlier point pairs and one or more parameters of a transform selected by the inlier detection. A second match score can be computed by applying the selected transform, and either or both scores can be used to authenticate the user.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a divisional of Ser. No. 15/906,331, filed on Feb.27, 2018, and entitled “Feature Extraction and Matching for BiometricAuthentication,” which is a continuation of U.S. patent application Ser.No. 15/259,902, filed on Sep. 8, 2016, and entitled “Feature Extractionand Matching for Biometric Authentication,” which is a continuation ofU.S. patent application Ser. No. 14/933,539, filed on Nov. 5, 2015, andentitled “Feature Extraction and Matching for Biometric Authentication,”which is a divisional of U.S. patent application Ser. No. 14/274,385,filed on May 9, 2014, and entitled “Feature Extraction and Matching forBiometric Authentication,” which claims the benefit of priority to U.S.Provisional Application Ser. No. 61/878,588, filed on Sep. 16, 2013, andentitled “Image Feature Detection, Authentication, and InformationHiding,” each of which is incorporated herein by reference in itsentirety.

TECHNICAL FIELD

The present disclosure generally relates to image comparison techniquesfor determining similarity between two images and, in particular, tobiometric authentication based on images of the eye.

BACKGROUND

It is often desirable to restrict access to property or resources toparticular individuals. Biometric systems can be used to authenticatethe identity of an individual to either grant or deny access to aresource. For example, iris scanners can be used by a biometric securitysystem to identify an individual based on unique structures in theindividual's iris.

In a typical image-based biometric authentication system, one or moreimages are obtained from a person claiming to be an authorized user.Those images are compared with one or more reference images that areknown to be provided by one or more authorized users. Only if theclaimant-supplied images match well with one or more reference images,the claimant can be determined to be an authorized user. A match betweentwo images, i.e., a measure of similarity between the two images, can bebased on features of those images. The features can include a collectionof one or more points of interest in the image, and descriptions ofportions of the image surrounding such interest points.

SUMMARY

This specification describes technologies that are well suited forbiometric authentication based on images of the eye. In particular, animage sharpening technique can aid in efficient feature detection. AVascular Point Detection (VPD) technique can detect the points ofinterest from visible vasculature of the eye, and Pattern Histograms ofExtended Multi-Radii Local Binary Patterns (PH-EMR-LBP) and/or PatternHistograms of Extended Multi-Radii Center Symmetric Local BinaryPatterns (PH-EMR-CS-LBP) can efficiently provide description of portionsof images surrounding a point of interest of vasculature. The visiblevasculature can be obtained using a scleral mask, which can be a binaryimage mask that includes white of the eye and excludes an image portionsurrounding the white of the eye from an ocular image. Matchingtechniques described herein can improve the efficiency and/or accuracyof distance or correlation based matching by using outlier detection.The described techniques also allow for updating templates that arederived from reference images and are used for authentication (generallyreferred to as enrollment templates in the discussion below), so thatthe best quality and diverse images are used in authentication, e.g., tominimize naturally occurring variances in images captured at differentlocations and times.

In a feature extraction and pattern matching system, image sharpeningcan enable vascular point detection (VPD) for detecting points ofinterest from visible vasculature of the eye. Pattern Histograms ofExtended Multi-Radii Local Binary Patterns and/or Pattern Histograms ofExtended Multi-Radii Center Symmetric Local Binary Patterns can providedescription of portions of images surrounding a point of interest, andenrollment and verification templates can be generated using pointsdetected via VPD and the corresponding descriptors. Inlier point pairscan be selected from the enrollment and verification templates, and afirst match score indicating similarity of the two templates can becomputed based on the number of inlier point pairs and one or moreparameters of a transform selected by the inlier detection. A secondmatch score can be computed by applying the selected transform, andeither or both scores can be used to authenticate the user.

An enrollment template can be a collection of interest points such asvascular points (VPD) and corresponding features such as EnhancedMulti-Radii Local Binary Patterns (EMR-LBP), Pattern Histograms ofEnhanced Multi-Radii Local Binary Patterns (PH-EMR-LBP), Patternhistograms of Enhanced Multi-Radii Center-Symmetric Local BinaryPatterns (PH-EMR-CS-LBP), and Enhanced Multi-Radii Center-SymmetricLocal Binary Patterns (EMR-CS-LBP). In some implementations, anenrollment template can be created only if the acquired image exceeds acertain threshold based on ratio of VPD points to that of size ofsegmented scleral region. More than one enrollments are possible for asingle user. Enrollment templates can be updated to accommodatebehavioral and/or environmental variations affecting the acquired scans.Updating the enrollment templates using verification can be based onquality of a candidate verification template, match score, and/or otherimage and exposure similarity measures.

Accordingly, in one aspect, a computer-implemented method includesobtaining a sharpened image based on a number of captured images. One ormore captured images can include images of a vascular structure.Moreover, the method includes detecting several vascular points in thesharpened image and, for each one of a number of detected vascularpoints, generating a one or more different local image descriptors. Themethod also includes generating a template that includes one or more ofthe detected vascular points and their respective local imagedescriptors. Other embodiments of this aspect include correspondingsystems, apparatus, and computer programs.

In one implementation, obtaining a particular sharpened image includesselecting one or more images from the several captured images andaveraging the selected images to generate an average image. The methodalso includes convolving the average image with a Laplacian of Gaussian(LoG) kernel to obtain a convolved image, and subtracting each pixel ofthe convolved image from a maximum pixel value, to obtain a differenceimage. In addition, this implementation of the method includesmultiplying, in a pixel-wise manner, the difference image and theaverage image, to obtain the particular sharpened image. In someimplementations, instead of using a LoG kernel, computing the differenceimage and multiplying the difference image and the average image, anumber of Gabor kernels oriented at different angles can be used forconvolving the average image, so as to obtain the sharpened imagedirectly.

In another implementation, the average image can be convolved with a setof even Gabor kernels to obtain the sharpened image. The set of evenGabor kernels that are oriented across various angles can be tuned basedon several parameters, such as the resolution and scale of the inputimage, and average width of visible vasculature. This convolved imagemaybe used instead of, or in addition to, above mentioned LoG-basedsharpened image.

In one implementation, detecting a number of vascular points includesselecting a point in the sharpened image as a candidate vascular point.Then, several (e.g., N where N is greater than one) first neighborhoodpoints according to first window centered on the candidate vascularpoint can be identified. In addition, several (e.g., N) other secondneighborhood points according to second, different window centered onthe candidate vascular point can also be identified. In thisimplementation, the method further includes determining N states where Nis greater than one. Each of the N states corresponds to oneneighborhood point in the N points of the first neighborhood of points.A state can be determined by performing a comparison based on, at leastin part, respective intensities of one of the first neighborhood points,i.e., one of the N points according to the first window, a correspondingpoint of the second neighborhood, i.e., one of the N points according tothe second window, and the candidate vascular point. The method can alsoinclude aggregating the N states, and designating the candidate vascularpoint as a vascular point based on, at least in part, a value of theaggregated states.

In some implementations, a geometric distance between the candidatevascular point and a first neighborhood point in the first window isless than a geometric distance between the candidate vascular point anda first neighborhood point in the second window, where the firstneighborhood point in the second window corresponds to the firstneighborhood point in the first window. The comparison can includetesting if an intensity of the point in the first neighborhood of Npoints is greater by a first threshold than an intensity of thecandidate vascular point. Alternatively, or in addition, the comparisoncan include testing if an intensity of the corresponding point in thesecond neighborhood of N points is greater by the first threshold thanthe intensity of the candidate vascular point. Determining acorresponding state can include setting the state to a first value(e.g., a logical high value or “1”) if any of the two tests is true, andsetting the state to a second value (e.g., a logical low value or “0”),otherwise. In one implementation, aggregating the N states includessumming the N states, and designating includes testing if a summed valueof the aggregated states exceeds a selected count. The selected countcan be N or, in some implementations, can be less than N.

Another comparison can include testing whether the intensity of anypoint in the first neighborhood of N points is greater than a secondthreshold, and/or testing whether the intensity of the correspondingpoint in the second neighborhood of N points is greater than the secondthreshold. Determining the corresponding state can include setting thestate to a second value (e.g., a logical low value or “0”) if any of thetwo tests is true.

In some implementations, the method includes performing the selecting,identifying, determining, aggregating, and designating for a number ofpoints in the sharpened image, and setting each candidate vascularinterest point designated as a vascular point to a first value (e.g., alogical high value or “I”) and setting other candidate vascular interestpoints to a second value (e.g., a logical low value or “0”), to obtain abinary vascular map (BVM) representing veins. The BVM can be thinned byexcluding at least one vascular point that: (i) corresponds to theboundaries across the width of a vasculature, and (ii) was set to thefirst value. The method can also include locally suppressing one or morevascular points that were set to the first value. The local suppressioncan be based on, at least in part, a gradient magnitude map relating toa portion of the sharpened image or relating to the entire sharpenedimage.

In some implementations, generating the respective one or more localimage descriptors includes computing at least one of: (i) a patternhistogram of extended multi-radii local binary patterns (PH-EMR-LBP) ofan image region surrounding the detected vascular point, and (ii) apattern histogram of extended multi-radii center-symmetric local binarypatterns (PH-EMR-CS-LBP) of an image region surrounding the detectedvascular point.

In another aspect, a computer-implemented method for matching one ormore verification templates with one or more enrollment templatesincludes identifying a number of matching point pairs. Each matchingpoint pair includes a first point from a particular verificationtemplate and a corresponding second point from an enrollment template.Each first point includes: (i) a location of a point of interest in averification image corresponding to the verification template, and (ii)a number of different types of descriptors, each one describing alocality surrounding the point of interest in the verification image.Each second point includes: (i) a location of a point of interest of anenrollment image corresponding to the enrollment template, and (ii) anumber of different types of descriptors, each one describing a localitysurrounding the point of interest in the enrollment image.

The method also includes obtaining several inlier point pairs selectedfrom the a number matching point pairs by performing outlier detectionacross the verification and enrollment templates. In addition, themethod includes calculating a match score based on the several inlierpoint pairs (e.g., a stage 1 match score), using a geometrictransformation identified during outlier detection as part ofpreprocessing (e.g., in computing a stage 2 match score), or both. Otherembodiments of this aspect include corresponding systems, apparatus, andcomputer programs.

In one implementation, identifying the several matched point pairsincludes, for each first point in the verification template, calculatingrespective distances between the descriptors associated with the firstpoint and descriptors associated with one or more of the second pointsof the enrollment template. One of the second points can be designatedas corresponding to the first point based on the respective distancesand, thus, a match point pair that includes the first point and thecorresponding second point is identified.

Calculating a respective distance between the descriptors associatedwith a particular first point in the verification template anddescriptors associated with a second point of the enrollment templatecan include calculating a distance between each descriptor associatedwith the particular first point and each corresponding descriptor of thesecond point of the enrollment template. The calculated distances can becombined as a weighted average to obtain the distance between thedescriptors associated with the particular first point and thedescriptors associated with the second point of the enrollment template.

In one implementation, identifying the several matching point pairsaccording to a voting method includes, for each first point in theverification template, calculating respective distances between each ofthe descriptors associated with the first point and a correspondingdescriptor associated with one or more second points of the enrollmenttemplate. A number of distances not exceeding respective distancethresholds can be counted, and one of the second points can bedesignated as corresponding to the first point based on the count of thenumber of distances. Thus, a match point pair including the first pointand the corresponding second point is generated.

In various implementations, calculating respective distances can becalculated as a Hamming distance, a Euclidean distance, a Manhattandistance, a correlation, or a Mahalanobis distance. In someimplementations, the local (non-binary) descriptors can be shortenedusing principal component analysis (PCA) to eliminate dimensions that donot contribute to local descriptor variances. A particular descriptorcan be derived using Extended Multi-Radii Local Binary Patterns(EMR-LBP), Histograms of Extended Multi-Radii Local Binary Patterns(H-EMR-LBP), Patterned Histograms of Extended Multi-Radii Local BinaryPatterns (PH-EMR-LBP), Extended Multi-Radii Center Symmetric LocalBinary Patterns (EMR-CS-LBP), Histograms of EMR-CS-LBPs (HCS-LBP),Pattern Histograms of EMR-CS-LBPs (PH-EMR-CS-LBP), Histograms ofOriented Gradients (HoG), Speeded Up Robust Features, (SURF) or FastRetina Keypoint (FREAK). Obtaining the several inlier point pairs caninclude using random sample consensus (RANSAC), M-estimator sample andconsensus (MSAC), or GROUPSAC, to align the first points to thecorresponding second points.

In some implementations, calculating the match score includes computingcorrelation of detected inlier points' locations from the matched firstpoints and corresponding second points. Computing the first match scorealso includes additional parameters and procedure besides locationcorrelation such as number of the inlier point pairs and one or moreparameters of the aligning geometric transformation. A parameter of thegeometric transformation can be a change in scale of the verificationimage as a result of the calculated geometric transformation from inlierpoint pairs or a change in angle of the verification image as a resultof the geometric transformation from inlier point pairs. In oneimplementation, both parameters are used in computing the first matchscore. In other implementations, depending on the transformation type,additional parameters such as shear maybe used. Other computed measuresof the geometric transformation showing its deviation from the genuinedistribution are also acceptable. In some implementations, the methodincludes, prior to identifying the candidate (inlier) points, modifyingthe averaged verification image via Gabor filtering. The enrollmenttemplate can also be modified via Gabor filtering.

In some implementations, computing the match score includes computing asecond match score by applying the geometric transformation to theverification image to create a transformed image. The method alsoincludes filtering the transformed image and encoding the oriented localbinary pattern versions of the transformed image, and applying the sameprocess to the image used for the enrollment template. Each version caninclude a number of layers wherein each layer corresponds to a distinctdimension of the encoding. The method can further include comparing oneor more corresponding tiles in each layer of the encoded transformedimage and the encoded image corresponding to the enrollment template, toobtain a respective layer measure for each layer. The layer measures canbe aggregated to obtain the second match score. The filtering caninclude Gabor filtering or logarithm of Gabor filtering, and theencoding can include Gray coding.

In some implementations, the method includes excluding, prior tocomparing, one or more tiles corresponding to a region substantiallylacking visible vasculature. Comparing corresponding tiles can includecalculating a Hamming distance, a normalized Hamming distance, or asliding window correlation between the corresponding tiles. The methodcan further include computing a first match score in addition to thesecond match score. The first match score can be computed by computing acorrelation between coordinates of the two corresponding points, fromthe verification and enrollment templates, respectively, of the inlierpoint pairs, and can be based on parameters such as a number of theinlier point pairs and one or more parameters of the geometrictransformation, and/or be based on a function of the number of inlierpoint pairs. The match score can then be computed as a weighted sum ofthe first match score and the second match score.

In one implementation, the correlation for first match score includescomputing

$\frac{{Cx} + {Cy}}{2},$here the coordinates of the inlier points across the verification andenrollment templates include the X and Y coordinates of each point inthe pairs, and Cx and Cy are correlations of the X and Y coordinates,respectively, of inlier matched points across the verification andenrollment templates. The Stage 1 match score can be computed as:

$\frac{\left( \frac{{Cx} + {Cy}}{2} \right)*{\log_{2}(N)}}{\left( {1 + {{\log_{2}\left( {{RS} + 0.001} \right)}}} \right)*\left( {1 + \left( \frac{RA}{0.2} \right)^{2}} \right)}$In this computation, N is the number of the inlier point pairs, RS is achange in scale of the verification image as a result of the calculatedregistration geometric transformation, and RA is a change in angle ofthe verification image as a result of the calculated registrationgeometric transformation. The method can further include excluding thefirst match score from the match score if at least one parameter of thegeometric transformation lies outside a nominal range of that parameter.RS and RA can be augmented by shear when affine transformation isassumed. The first stage match score, in terms of a function of Cx, Cy,N, and the transformation matrix-derived parameters, can also bedirectly learned using labeled dataset of impostor and genuinecomparisons to train a classifier such as an artificial neural networkor linear discriminant analysis. Optionally, a PCA preprocessing stagemaybe applied before classification.

In some implementations, instead of transformation-matrix-derivedparameters RS and RA, another function of the transformation can bedirectly calculated from the its matrix elements. Considering thetransformation matrices derived from genuine and impostor comparisons,it is desired to create a function of corresponding transformationmatrix elements yielding maximum genuine-impostor separation in itsdistribution. One way to achieve this end is to use the transformationmatrices of a labeled dataset of impostor and genuine comparisons totrain a regression function and maximize a measure of classifiability,such as Fisher discriminant ratio.

In another aspect, a computer-implemented method for updating enrollmenttemplates includes receiving a verification template that includesseveral points of interest. Each point of interest is associated with anumber of different respective descriptors that each describes one ormore localities surrounding the corresponding point of interest. Themethod also includes computing a match score for the verificationtemplate by comparing the verification template with one or moreenrollment templates in a collection of enrollment templates. Theverification templates can be added to the collection of enrollmenttemplates based on, at least, the match score matching or exceeding anenrollment threshold.

In some implementations, the method includes, for each template in thecollection of enrollment templates, generating a respective match scorewith one or more other templates in the collection of enrollmenttemplates. A respective median match score is computed for the template,and a template having a minimum median match score can be removed fromthe collection of enrollment templates. Other embodiments of this aspectinclude corresponding systems, apparatus, and computer programs.

In some implementations, the method may include generating quality scorefor enrollments using thinned BVM and the scleral mask. The scleral maskcan be a binary image mask that includes white of the eye and excludesan image portion surrounding the white of the eye from an ocular image.Quality score may be the ratio of number of detected vascular points inthinned BVM to number of true pixels (1's) in scleral mask. The methodalso includes removing enrollment templates which do not pass a certainquality score.

In some implementations, the method includes adjusting the match scoreaccording to an exposure difference, an influence threshold, or both.The exposure difference can include a difference between a verificationexposure template associated with the verification template and anenrollment exposure template associated with the enrollment template.The method can also include generating the verification exposuretemplate and/or the enrollment exposure template.

In some implementations, generating an exposure templates (forenrollment and/or verification) includes partitioning an ocular regionof interest (ROI) corresponding to the enrollment and/or verificationimages into two or more sections, and generating for each section ahistogram of intensities representing for each intensity in thehistogram, a number of pixels in the section of the ROI havingsubstantially that intensity. The ROI can be an area of the imagecentered on the eye. In some implementations, this area is found by aneye finder. In some implementations, the ROI is found by cropping animage of an eye according to a bounding box of a scleral mask and bypadding the cropped image with a specified number of pixels (e.g., 50pixels). In one implementation, the ROI is partitioned into fourquadrants. In some implementations, an image of an eye can be croppedusing a bounding box from one eye corner to another eye corner and fromone eye lid to another eye lid. The cropped image can be padded with aspecified number of pixels.

In one implementation, the exposure difference is −1 times an exposuresimilarity. The method includes determining for each quadrant j: one ormore of (i) a normalized absolute value of a histogram differences(ABSNdist_j), (ii) a histogram intersection similarity (INTRsim_j),(iii) a correlation coefficient similarity of histograms (CORRsim_j),and (iv) a Bhattacharyya distance (BHATdist_j). The exposure similaritycan be computed as:

-   -   (−Σ_(j) ABSNdist_j−Σ_(j) BHATdist_j+Σ_(j) INTRsim_j+Σ_(j)        CORRsim_j).        Other linear or nonlinear combinations of the above four metrics        are possible.

In one implementation, generating one or more exposure templatesincludes generating a number of exposure measures. Each one of theexposure measures can be an exposure metering of the ROI included inEXIF file of the image (EXIF measure), a statistical parameter of a Ycomponent of a YUV image, and a statistical parameter of a G componentof an RGB image.

In some implementations, an exposure difference and/or an exposuresimilarity is computed based on one or more histogram measures. Thedifferent histogram measures can be any one of: a normalized absolutevalue of a histogram differences (ABSNdist), a histogram intersectionsimilarity (INTRsim), a correlation coefficient similarity of histograms(CORRsim), and a Bhattacharyya distance (BHATdist).

In some implementations, the method may include, prior to computing thematch score, ordering the collection of enrollment templates accordingto respective exposure differences between a verification exposuretemplate associated with the verification template and each enrollmentexposure template associated with each enrollment template in thecollection of enrollment templates. As such, the verification of theverification template can be expedited because generally theverification proceeds by comparing a verification template with severalenrollment templates, but starting with those that have lowest exposuredifference and, thus, more likely to correctly match the verificationtemplate.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1A schematically depicts an example two-stage system for generatingenrollment templates.

FIG. 1B schematically depicts an example two-stage system for generatinga verification template and comparing that template with an enrollmenttemplate, thereby determining the similarity between the two imagescorresponding to the two templates.

FIGS. 2A-2D illustrate an example image sharpening to aid vascular pointdetection and local image descriptor generation.

FIGS. 2E and 2F illustrate an example vascular point detectionprocedure.

FIGS. 3A-3C depict, respectively, an example binary image of detectedvascular points from an eye, a corresponding thinned binary image, andan overlay of the thinned and the original binary images.

FIGS. 4A and 4B depict example masks used to sparse a thinned binaryimage representing detected vascular points.

FIG. 4C depicts a sparse binary image corresponding to the thinnedbinary image depicted in FIG. 3C.

FIGS. 5A-5C respectively depict an example original image of an eye, anexample region of interest (ROI) mask, and the ROI from the originalimage.

FIGS. 5D-5F respectively depict a binary vascular map (BVM)corresponding to the ROI depicted in FIG. 5C, a corresponding thinnedBVM, and a corresponding sparse BVM, each of which is overlaid on theoriginal image depicted in FIG. 5A.

FIG. 6 depicts a Gabor filtered oriented local binary patterns (OLBP)image corresponding to an example eye vasculature.

FIGS. 7A-7C illustrate generating Gray coded OLBP patterns correspondingto an example pixel window.

FIG. 8 illustrates an example process of updating enrollment templateson a rolling basis.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

FIGS. 1A and 1B schematically depict an example multi-stage biometricsystem for generating and storing enrollment templates of visiblevasculature that can be used for user authentication. In oneimplementation, the visible vasculature corresponds to the vasculatureseen in the white of the eye. The white of the eye has a number oflayers. The sclera is an opaque, fibrous, protective, layer of the eyecontaining collagen and elastic fiber. The sclera is covered by theconjunctiva, which has a particularly large number of blood vessels andveins that that run through and over it. The episclera is covered by thebulbar conjunctiva, which is a thin clear membrane that interfaces withthe eyelid or the environment when the eyelid is opened. Blood vessels(vasculature, in general) run through all of these layers of the whiteof the eye and can be detected in images of the eye. During userauthentication, one or more images of a user's eye are captured, one ormore verification templates are generated from the captured image orimages, and the identity of the user can be verified by matching thecorresponding vascular structures as expressed in the enrollment andverification templates.

It should be understood that the systems and methods described withreferences to FIGS. 1A and 1B are not limited to eye vasculature. Forexample, biometric authentication can be performed using vasculature ofthe ear. In particular, using a near infrared camera (for instance ifone is implemented near earpiece of a phone) and an infrared lightsource, images of the ear vasculature can be obtained and userauthentication using those images can be performed as described herein.

In step 102, images of a user's eyes are captured using an image sensor,e.g., a camera, that is associated with a device to which access by theuser is to be authenticated (e.g., a smart phone, a smart watch, smartglasses, a notebook computer, a tablet computer, etc.). By way ofillustration, the camera can be a digital camera, a three-dimensional(3D) camera, or a light field sensor. In some implementations, thecamera can be an inward facing module in a wearable device with aspectacle form factor, and used to capture images of the user's white ofthe eye for biometric authentication. The images can be captured eitherin still mode or in video mode. In some implementations, the user may beprompted (by visual, or audible, or haptic feedback) to look to the leftor right or up or straight while the image is captured. Looking to theleft or right can expose a larger area of the white of the eye to theright or left of the iris for data capture, while looking straight canprovide two smaller segments of the white of the eye to the left andright of the iris in each eye. In step 104, a general region of interestis identified, e.g., one or both eyes may be located. To this end,Viola-Jones algorithms trained on different glancing eye images can beused. Thereafter a gaze tracking algorithm such as that using Haarfilters can be used to quantize the amount of gaze to acquire one ormore images, once a selected gaze direction is detected. Typically, theacquired images are cropped to obtain RGB images of one or more eyes.Hereafter, “image” or “captured image” generally refers to a cropped RGBimage of an eye.

The images that are captured from an image sensor can have varyingquality due to, for instance, exposure and motion blur artefacts. Inoptional step 106, several images can be registered (i.e., spatiallyaligned) and averaged to reduce image noise. In some implementations,image correlation methods are used to measure the dissimilarity betweenthe obtained images in order to align the images for averaging anddiscard those that are most different (e.g. due to motion blur or eyeblink) and, hence, are not suitable for registration and averaging. Forexample, four images captured in sequence can be averaged afterregistration if the images are not too different. The number of imagesthat are averaged can depend on the frame rate and noise level of theimage sensor. When several averaged images are generated, a qualitymetric can be used to select the highest-quality for use in subsequentsteps. In some implementations, standard deviation of the pixel valuesof the green channel of a sharpened image (derived as described below)can be used as a quality metric. Other quality metrics are possible.

The images produced by the quality check step 106 are used to producetemplates for a two-stage matcher, after which the original images areusually discarded for security and privacy reasons. A stage 1 templatecan include a number of (interest point, feature vectors) elements. Astage 2 template can include encoded (e.g., Gray coded) oriented localbinary patterns (OLBP) corresponding to the original images. In anenrollment mode (described below), one or more captured images aredesignated as enrollment images and enrollment templates are generatedtherefrom. In some implementations, corresponding exposure templatesbased on local histograms of exposure profiles of the captured imagesare generated and stored. In an authentication/verification mode (alsodescribed below), one or more captured images are designated asverification images, and verification templates are generated therefromto match against enrollment template.

With reference to FIG. 1A, in Stage 1, one or more averaged images,which can be enrollment images or verification images, are preprocessedat step 108, and features from the preprocessed images are extracted atstep 110, to generate one or more Stage 1 templates. In Stage 1, thepreprocessing can include image sharpening or Gabor filtering, and thefeature extraction can include vascular point detection (VPD) andfeature descriptors (PH-EMR-LBP, PH-EMR-CS-LBP, SURF, EMR-LBP, HoG andFREAK), and their PCA preprocessing, all of which are described indetail below. In Stage 2, one or more averaged images are preprocessedat step 112, and features from the preprocessed images are extracted atstep 114, to generate one or more Stage 2 templates. In Stage 2, thepreprocessing can include Gabor filtering and the feature extraction caninclude detection of oriented local binary patterns (OLBP), for example.The enrollment templates, prior to storing, can be encrypted at step116.

In the verification mode, i.e., when verification images are processed,Stage 1 at step 110 includes identifying candidate interest points andfeature descriptors that are matched with enrollment template to derivea geometric transform (e.g., a affine transformation, or self-similaritytransformation) from detected inliers and computing a first match scorebased on one or more parameters of the identified transform and inliercoordinates. In Stage 2 at step 114 includes transforming thepreprocessed verification image using the identified geometrictransformation, Gabor filtering and phase thresholding, and encoding theresult with OLBP to get the stage 2 verification template, and computinga second match score using correlation of the tiled OLBP layers ofverification and enrolled templates. More than one first match score canbe computed by repeating the step 110, each repetition corresponding toa different geometric transform that is identified in that repetition.The one or more first match scores and the second match score can becombined in step 118 (FIG. 1B), and the verification image or images canbe determined to be authentic based on the combined score. In someimplementations only the first match score or scores are used indetermining authenticity while in other implementations, the first matchscore may be discarded based on one or more parameters of thecorresponding geometric transformation. In some implementations, morethan one first and second match scores can be computed by repeating thesteps 110 and 114. The one or more first match and second match scorescan be combined in step 118 (FIG. 1B), and the verification image orimages can be determined to be authentic based on the combined score. Insome implementations, the combination rule can be the max rule, theproduct rule, or the sum rule. In some implementations, the combinationrule can be a linear mixture determined by linear discriminant analysis.

Specifically, as depicted in FIG. 1B, Stage 1 processing and analysisincludes determining correlation between corresponding locations inenrollment and verification templates, comparing image descriptors ofthe enrollment and verification templates that describe localitiessurrounding detected points of interest in images corresponding to thosetemplates to find a spatial correspondence of the points of interest,and rejecting outlier points of interest. At this stage, a registrationtransformation matrix, plus a match score is produced. Thetransformation matrix can be used to register the incoming verificationimages for the Stage 2 matcher. In some implementations, Stage 1 matchscore is used by itself to make a decision as to whether theverification image is authentic without proceeding to Stage 2. In someimplementations, if stage 1 fails to compute or fails to provide aregistration matrix within the acceptable scale, rotation, and/or shearlimits, stage 2 may proceed to produce a match score on its own using noor an auxiliary registration matrix provided by, for instance, step 104,based on the boundaries of the iris within the ROI.

Image Sharpening

In some implementations, preprocessing (step 108) can be used to enhancethe visibility of vasculature in the captured images. One such method isselecting a color component from the RGB image data that can maximizethe contrast between vasculature and surrounding white of the eye. Insome implementations, this image can include of a linear combination ofblue and green layers. Specifically, in one implementation, eachaveraged image is sharpened to obtain a corresponding sharpened imageusing a three-step process. The first step includes convolving the pixelvalues of green channel or part of the blue and green channels of anaveraged image derived from one or more captured images with a Laplacianof Gaussian (LoG) kernel, to obtain a convolved image. The second stepincludes subtracting the LoG filtered image from the maximum valueattainable in the image.

Finally, in the third step each pixel of the processed image obtainedfrom the second step is multiplied with the corresponding pixel of theaveraged image, to obtain a sharpened image. FIG. 2A, for example,depicts an averaged image of an eye obtained from one or more capturedimages. FIG. 2B depicts the corresponding green channel, and FIG. 2Cdepicts the corresponding LoG filtered image. The standard deviation ofthe sharpened imaged obtained in the third step can be the image qualitymetric used in sorting a sequence of averaged images and selecting anaveraged image for further processing. In one implementation, a LoGkernel used for sharpening can be expressed as:

${{LoG}\mspace{14mu}{Kernel}} = {{- \frac{1}{{\pi\sigma}^{4}}}\left( {1 - \frac{x^{2} - y^{2}}{2\sigma^{2}}} \right)\exp^{- \frac{x^{2} - y^{2}}{2\sigma^{2}}}}$where the filter kernel size is 5×5 pixels with a Gaussian standarddeviation (σ) of 0.4. Parameters of the LoG filtering can be optimizedbased on the resolution of the acquired image. The above exampleparameters were selected for a cropped eye image having approximatedimensions of 100×110 pixels (±20 pixels). The number of retainedquality sorted sharpened averaged images can vary according to theapplication. For example, if the application is generating and storingenrollment templates, the top four sharpened averaged images may beretained. A different number of images, such as two, may be retained ifthe application is at the verification stage.

In some implementations, in addition to the LoG-based image sharpening,another enhanced copy of the averaged image can be obtained byconvolving with bank of Gabor filters defined by:

${G_{even}\left( {x,y,f,\theta} \right)} = {\exp\left\{ {\frac{- 1}{2}\left\lbrack {\frac{x^{\prime\; 2}}{\sigma_{x}^{2}} + \frac{y^{\prime\; 2}}{\sigma_{y}^{2}}} \right\rbrack} \right\}\;{\cos\left( {2\pi\;{fx}^{\prime}} \right)}}$${G_{odd}\left( {x,y,f,\theta} \right)} = {\exp\left\{ {\frac{- 1}{2}\left\lbrack {\frac{x^{\prime\; 2}}{\sigma_{x}^{2}} + \frac{y^{\prime\; 2}}{\sigma_{y}^{2}}} \right\rbrack} \right\}{\sin\left( {2\pi\;{fx}^{\prime}} \right)}}$where x′=x sin(θ)+y cos(θ) and y′=x cos(θ)−y sin(θ) σx and σy define thestandard deviation of the Gaussian envelope along x-axis and y-axisrespectively, f is the frequency of the modulating sine or cosine, and θis the orientation angle of the kernel, varied 6 times, from θ to 5π/6,each π/6 radians apart. The resulting six filters are used for aforesaidimage enhancement. In some implementations, only the magnitude of theodd or even part of the Gaussian filter bank may be used. In someimplementations, f is 0.22 and σ_(x)=σ_(y)=1.9. These values can changewith the resolution of the ROI. In one implementation, only even Gaborfilters are used to enhance averaged image.

In some implementations, additional preprocessing of the sharpenedaveraged images includes image histogram and contrast adjustments suchas Contrast Limited Adaptive Histogram Equalization (CLAHE). CLAHEgenerally operates in small regions of the image called tiles.Typically, each tile's contrast is enhanced such that the histogram ofthe output approximately matches the histogram specified by a particulardistribution (e.g., uniform, exponential, or Rayleigh distribution). Theneighboring tiles are then combined using an interpolation (e.g.bilinear interpolation) to eliminate any artificially inducedboundaries. In some implementations, the image region can be enhanced byselecting a linear or nonlinear combination of the red, green, or bluecolor components that has the best contrast between the vessels and thebackground. For example, the green component can be preferred in a RGBimage of the eye, because it can provide the best contrast betweenvessels and the background. In some implementations, green and partialblue pixel values can be used. In some implementations, CLAHEpreprocessing is not used. A Region of Interest (ROI) can be selectedfrom the averaged image prior to the three-step sharpening, for example,by applying a scleral mask. A check can be used to ensure the soundnessof the selection of the ROI. For example, if the area selected byapplying the scleral mask is not at least a certain specified percentage(e.g., about twenty five percent of the averaged image size), thecorresponding image is removed from further processing.

Vascular Point Detection

Various point detection algorithms can be used to identify salient orinterest points within a region of interest (step 104). Salient orinterest points are typically where potentially identifiable vascular orotherwise specific patterns of interest generally occur. Some knowntechniques that can be employed for interest point detection include theSpeeded Up Robust Features (SURF) algorithm, Accelerated Segment Test(FAST) algorithm, and Harris and Stephens (HS) algorithm. These generictechniques often do not detect most/all the points on the vascularpatterns within an image region. Therefore, in some implementations, aVascular Point Detection (VPD) algorithm is used. The VPD is a pointfinder that tends to find points that are centered/located onvessel-like objects. The VPD algorithm considers distal neighborhoods ofa potential interest point (pixel) of an intensity image to determine ifthat point is located on an object of interest, e.g., a vessel in avascular pattern. The VPD algorithm can be adjusted based on, forexample, the scale and resolution of an image, pixel intensity dynamics,and a specified point detection sensitivity, among other factors.

In one implementation of the VPD algorithm a region of an image isselected, as depicted in FIG. 2D. Various distal neighborhood points, asdepicted in FIG. 2E are considered for VPD calculations. With referenceto FIG. 2F, in an example pixel neighborhood centered on an interestpoint candidate P0 two pattern windows 202, 204 are identified. In someimplementations, additional pattern windows can be embedded within eachpixel neighborhood. Each pixel neighborhood is laid on potential pointof interest (e.g., P0). The sizes of the two pattern windows 202, 204are different. In one implementation, shown in FIG. 2F, the firstpattern window is a 5×5 pixel kernel which includes points P1 throughP8, while the second pattern window is a 7×7 pixel kernel which includespoints P1′ through P8′. Each pattern window includes eight peripheralpixels that are used for computations described below.

To determine whether the center point P0 is a point of interest, the VPDcalculates the difference in intensities between candidate pixel (P0)and 16 other pixels (i.e., P1-P8 and P1′-P8′) in the neighborhood of thepixel P0. Each comparison is considered to be a state and each state isdefined as:Si=((Pi−P0)>t)∥((Pi′−P0)>t)

Specifically, the intensity of P0 is compared with intensities ofcorresponding edge pixels Pi and Pi′. If the difference between theintensities of P0 and Pi exceeds an intensity threshold t, or if thedifference between the intensities of P0 and Pi′ exceeds the intensitythreshold t, the state is set to a high logical value (e.g., “1”).Otherwise, the state is set to a low logical value (e.g., “0”). Theintensity threshold t can be changed based on the quality of images andthe amount of noise in the images. In this example, a maximum countervalue is eight, as there are eight comparisons. If the sum of all eightstates exceeds a certain count, e.g., 6, the candidate point is labelledor designated as a vascular point.

In some implementations, if an intensity value of any pixel in a windowis greater than a second intensity threshold, the center point can bediscarded from further analysis. In this situation, the pixel mayrepresent an area of the image where aberrations, glare, or other imageartifacts may lead to a faulty identification of a candidate point. Forexample, if a second intensity threshold is 240, where the maximumintensity of 256, and if the intensity of an edge pixel exceeds 240, thecorresponding center pixel is discarded from a pool of potential VPDcandidate points.

The values provided above are example values, and other values can beused. For example, more than two windows can be used, the sizes ofeither or both windows can be different than those in this example, andthe threshold values and intensity scale can also be adjusted based onvarious factors. For example, an image with relatively low resolutioncan use two small pixel windows, such as one 3×3 window and one 5×5window, while a relatively high resolution image can use three largerpixel windows, such as one 7×7 window, one 9×9 window, and one 11×11window. The VPD process can be used on all pixels of an image region, ora subset of pixels, such as each pixel within a particular distance fromthe border of the image region. In general, the VPD is a localneighborhood operation and a sliding window technique can be employed indetermining the states as described above and in deciding whether acandidate point is a vascular point. As such, the VPD binarizes the ROIof an eye by assigning to all points determined to be vascular points alogical high value (e.g., “1”), and by assigning to all other candidatepoints a logical low value (e.g., “0”). The resulting binary mask isreferred to as Binary Vascular Map (BVM) as depicted in FIG. 3A.

In a BVM, all blobs that are less than a predefined number of pixels insize can be removed by considering them to be non-vascular points. Insome implementations, each individual connected vascular structure(blob) obtained from a BVM can be thinned, resulting in a thinned BinaryVascular Map or thinned BVM, as depicted FIG. 3B. The thinning processcreates a vascular trace traversing through the center of vasculature.An overlay of an example thinned BVM on a corresponding BVM is shown inFIG. 3C.

In some implementations, in order to further reduce the number ofvascular points, a local point suppression can be applied. In general,local suppression is based on gradient magnitude map obtained from agray scale sharpened image of an ROI. The gradient magnitude map can beobtained by convolving a Sobel filter with the gray scale image toemphasize edges. The Sobel operator generally convolves the ROI imagewith 3×3 horizontal and vertical gradient kernels to yield gradientmagnitude mask. Example gradient kernels are shown in FIGS. 4A and 4B.

In one implementation, a thinned BVM is divided into neighborhoods of5×5 non-overlapping blocks. In each block, if any thinned BVM points arepresent, only one vascular point that maps to highest gradient value inthe corresponding gradient magnitude mask is chosen. This process oflocal suppression reduces the number of vascular points by almost half,thus reducing the template size and aiding the process of matching. Thereduced set of interest points is referred to as “sparse VPD points”hereafter. An example set of sparse VPD points corresponding to theexample BVM depicted in FIG. 3B is shown in FIG. 4C. The 5×5neighborhood is illustrative only, and can be changed based on the imageresolution and scale. FIGS. 5A-5F depict an example original image, anexample ROI mask, the ROI from the original image, a BVM correspondingto the ROI overlaid on the original image, a corresponding thinned BVMoverlaid on original image, and a corresponding sparse BVM (also calledsparse VPD) overlaid on the original image.

In one implementation, candidate points from other point finders such asFAST, HS, and SURF can be added to the sparse VPD set of points,provided that they satisfy a minimum distance threshold. For example, inone implementation, FAST points are added to the sparse VPD points ifthe FAST points are at least 3 pixel away from the VPD points. In someimplementations, interest points can be derived from all or a subset ofthe above-described interest point finders. In some implementations,interest points can be identified at multiple scales. For example,interest points can be detected from a three stage Gaussian imagepyramid. Other multi-scale image decompositions are possible.

Local Image Descriptors

After interest points are identified using one or more point detectionalgorithms described above, a set of one or more local image descriptorscan be obtained from the ROI localities surrounding each candidate point(step 110). These local image patch descriptors can be generated using avariety of algorithms such as histograms of oriented gradients (HoG) andSpeeded Up Robust Features (SURF) which builds upon SIFT descriptors butwith better computational efficiency using through Haar wavelets andintegral images. Local image descriptors can also be computed using abinary local image descriptor algorithm called Fast Retina Keypoint(FREAK). Extended Multi-Radii Local Binary Pattern (EMR-LBP) and PatternExtended Multi-Radii Center-Symmetric Local Binary Patterns(PEMR-CS-LBP) are two other binary image feature extractors. In general,these techniques are not optimized for eye vein matching.

Pattern Histograms of EMR-LBP (PH-EMR-LBP) and Pattern Histograms ofEMR-CS-LBP (PH-EMR-CS-LBP) as feature descriptor algorithms, describedin further detail below, are customized for eye-vein matching. Each ofthese techniques can be used individually, in combination, and/or incombination with other feature descriptor algorithms described above togenerate several image descriptors around each interest points of anROI.

Generally, the LBP descriptors are calculated around interest points asfollows: it is assumed that the current interest point is at pixellocation (x₀,y₀). The intensity values of the immediate 8-neighbors ofthe center point (x₀,y₀), {(x₁,y₁)}, i=1, 2, . . . 8), are compared tothat of the center point and the results are stored in K_(i). Theintensity of a neighbor can be less or the same as the intensity of thecenter point, and the corresponding result can be a logical low valuesuch as “0.” If the intensity is greater, the result can be a logicalhigh value such as “1.” After the comparisons, and 8-bit code calledLBP8 for (x₀,y₀) is obtained as

${{LBP}\; 8} = {\sum\limits_{i}{K_{i}*2^{i - 1}}}$

In one implementation, the above-described process can be repeated forthe pixels in the outer square of LBP8, thus generating a 16-bit (2byte) LBP16 code for a particular interest point. Thus a total of 3bytes of LBP code for each interest point can be generated. The processcan be repeated for 5×5 pixel neighborhood of (x₀,y₀), yielding a totalof 5×5=25 repetitions of the above LB8(1 byte)+LBP16(2 bytes)calculations surrounding a particular center point, resulting in a3×25=75 byte binary descriptor for each center interest point such as(x₀,y₀). This 75 byte binary descriptor is designated as ExtendedMulti-Radii Local Binary Patterns (EMR-LBP). Windows are not confined totwo; other sizes are possible based on image resolution.

In a EMR-CS-LBP-based implementation, the reference point for eachpairwise pixel comparison includes the diagonally opposite pixel in an8-pixel or 16-pixel neighborhood of the center point of interest,instead of the intensity value of the center point, thereby leading tohalf the number of bits compared to EMR-LBP features. Both EMR-LBP andEMR-CS-LBP descriptors are binary numbers.

HoG is typically calculated using a neighborhood of a certain size (inpixels) defined around an interest point. That neighborhood can bedivided into a predefined number of sub regions, within which histogramsof edge orientations at certain angles can be created and collectivelyused as the local descriptor for that interest point. Thesehistogram-based descriptors are real number vectors. In oneimplementation, a neighborhood of size 4×4 pixels is tiled into 2×2 subregions with histograms of orientations binned into 6, each 30 degreesapart, and are used as feature descriptors. In one implementation, aneighborhood of size 4×4 pixels tiled into 3×3 sub regions withhistograms of orientations binned into 6 bins (each 30 degrees apart)can be used as feature descriptors.

HEMR-LBP descriptors can also be calculated using a neighborhood of acertain size (in pixels) around the interest point. EMR-LBP codes (asdescribed above) for each pixel of the neighborhood are derived. Thatneighborhood of EMR-LBP codes is divided into a predefined number of subregions. Next, to create histograms, counts for occurrence of each bitlocation within a sub-region are generated. The concatenations of thesehistograms of EMR-LBP codes across all the sub-regions can be designatedas PH-EMR-LBP features. These descriptors are real number vectors. Invarious implementations, a neighborhood of size m×m pixels (m=4, 5, . .. , 11, etc.) is tiled into n×n (n=2, 3, . . . , 7, etc.) overlappingsub regions (tiles), and the concatenation of histograms of occurrenceof each EMR-LBP bit location within a neighborhood or a sub-region canbe used as feature descriptors. The choice of m and n can made based onthe obtained image resolution. HEMR-LBP are similar PH-EMR-LBP but hasno sub-regions as the histograms are calculated on the entireneighborhood. HLBP are similar to HEMRLBP but uses just a LBP.

In one implementation, a neighborhood of size 9×9 pixels (whose LBP8 andLBP16 codes are already generated, as described above) are tiled intosixteen 3×3 sub regions, each with one pixel overlap. Each 3×3sub-region of LBP8 codes is converted to a string of nine unsigned 8-bitnumbers, similarly LBP16 are converted to 9 unsigned 16-bit numbers. Theunsigned 8-bit numbers of LBP8 strings are converted to unsigned 16-bitnumber strings. Histograms of occurrence of each bit location of 9unsigned 16-bit strings are calculated, each can deliver a vector oflength 16 bits. Thus, each sub region can have two vectors of length 16from LBP8 and LBP16 codes that are concatenated to deliver a finallength of 512 unsigned 16-bit numbers using 16 sub-regions (PH-EMR-LBPdescriptor of the image patch).

PH-EMR-CS-LBP can be calculated using a neighborhood of a certain size(in pixels) around the candidate point. After generating EMR-CS-LBPcodes for each pixel in the neighborhood, that neighborhood is dividedinto a predefined number of sub regions. Next, counts for occurrence ofeach bit location within a sub-region are generated. The concatenationsof these histograms of EMR-CS-LBP codes can provide the PH-EM R-CS-LBPfeatures. These descriptors are real numbers. A neighborhood of size m×mpixels (m=4, 5, 6, 7, 8, 9, 10, 11, etc.) can be tiled into n×n subregions (n=2, 3, 4, 5, 6, 7, etc.), that can have overlapping tiles. Thehistograms of occurrence of each bit location within a neighborhood or asub-region can be used as feature descriptors. The choice of m and n canbe made based on the obtained image resolution. HEMR-CS-LBP are similarto PH-EMR-CS-LBP but has no sub-regions as the histograms are derived onthe entire neighborhood.

In one implementation, a neighborhood of size 7×7 pixels (whose CS-LBP8and CS-LBP16 codes are already generated as described above) are tiledinto nine 3×3 sub regions, each with one pixel overlap. Each 3×3sub-region of CS-LBP8 and CS-LBP16 codes is converted to string of nineunsigned 8-bit numbers for each of CS-LBP8 and CS-LBP16 code. Histogramsof occurrence of each bit location are calculated that can yield 8 binsfor CS-LBP8 and 8 bins for CS-LBP16. Concatenating all of the ninesub-regions can yield a vector length of 144 unsigned 16-bit numbers(PH-EMR-CS-LBP descriptor of the image patch).

In some implementations, feature descriptors for the image patch aroundan interest point can be derived from a single feature descriptoralgorithm, or using a number of different feature descriptor algorithmsdescribed above. For example, one or more of the following descriptorscan be used to characterize the image patch around each interest pointfor the purposes of creating a stage 1 template: EMR-LBP, CS-LBP, HoG,SURF, PH-EMR-LBP, and PH-EMR-CS-LBP. In some implementations, featuredescriptors can be derived around candidate points at multiple imagescales (yielding multi-scale feature extraction). For example, one maydetect interest points and their corresponding local image descriptorsusing a three-stage Gaussian image pyramid. Other multi-scale imagedecompositions are possible.

Verification or Authentication Via Matching

In general, matching is a process of finding similarity between one ormore saved enrollment template(s) associated with a user against one ormore verification template(s) of a claimant of that identity. If thesimilarity of claimant's verification template with the enrollmenttemplate (which can be expressed as match score) exceeds a specifiedthreshold, the claimant can be verified as an authenticated user.Otherwise, the claimant can be rejected.

Stage 1 Pattern Matching

A stage 1 template, created at the time of enrollment or verification(step 110), can include derived interest points and the correspondingimage feature descriptors around them. A Hamming distance is calculatedfor binary descriptors (FREAK, EMR-LBP and PEMR-CS-LBP) in order to findthe best matched point pairs between enrollment and verificationtemplates. The lower the Hamming distance the more similar the comparedpoints. For real valued descriptor vectors, Euclidean, Manhattan,correlation, or Mahalanobis distance between SURF, HoG, PH-EMR-LBP andPH-EMR-CS-LBP descriptors of an enrollment template and the respectiveSURF, HoG, PH-EMR-LBP and PH-EMR-CS-LBP descriptors of the verificationtemplate can be computed to determine if one or more of those distancessatisfy the specified, corresponding thresholds. Other distance measurescan also be used in determining matching point pairs. In particular, oneor more of the following histogram distance or similarity metrics can beused for matching histogram-based image descriptors such as PH-EMR-LBPand PH-EMR-CS-LBP: (i) a normalized absolute value of a histogramdifferences (ABSNdist_j), (ii) a histogram intersection similarity(INTRsim_j), (iii) a correlation coefficient similarity of histograms(CORRsim_j), and (iv) a Bhattacharyya distance (BHATdist_j).

Generally, a stage 1 template includes a set of t_(i) (interest point,feature vectors) elements in the following format:T={t _(i) },t _(i)=[(x _(i) ,y _(i)),{right arrow over (V)} _(i) ¹,{right arrow over (V)} _(i) ² , . . . ,{right arrow over (V)} _(i)^(d)],i=1,2, . . . ,n(T)where (x_(i), y_(i)) is the location of interest point i, and [{rightarrow over (V)}_(i) ¹, {right arrow over (V)}_(i) ², . . . , {rightarrow over (V)}_(i) ^(d)] is a collection of d different types ofdescriptor vectors that describe the local image patches around thepoint of interest at pixel coordinate (x_(i), y_(i)).

In some implementations, the matcher performs an exhaustive search (step110), calculating the distances between each feature vector {right arrowover (V)}_(i) ^(j) associated with a point of interest i, for allinterest points in enrollment template and all corresponding featurevectors of all points of interest in the verification template, toselect one or more matching point pairs. In some implementations, aratio test is implemented so that vague correspondences, with a first tosecond closest match distance ratio larger than a certain threshold, arediscarded as ambiguous katches. In other implementations, a kd-treebased technique can be used to match features using nearest neighboralgorithms implemented by Fast Library for Approximate Nearest Neighbors(FLANN) matcher. This can enable faster nearest neighbor searches amonghigh dimensional data points.

In some implementations, a voting method across all or a subset ofdescriptors V_(i) ^(k), k=1, 2, . . . d, can be used to select thecorresponding matching points from the enrollment and verificationtemplates. For example, one or more points from one template are pairedwith the corresponding one or more points from the other template onlyif a majority of the corresponding local image descriptors satisfy thedistance threshold. The voting method can be used when each type ofdescriptor, by itself, may not reveal the same set of matched pointpairs. Therefore, in one example, if the points in the two templates arematched using any three (at least) out of a total of five types ofdifferent descriptors the corresponding points in the verificationtemplate are considered as matched with the corresponding points in anenrollment template. Specifically, in one example, for each point ofinterest if the templates use five types of descriptors, namely,EMR-LBP, PH-EMR-LBP, PH-EMR-CS-LBP, HoG, and SURF descriptors, aninterest point can be considered to be a candidate for a matched pointpair if PH-EMR-LBP, PH-EMR-CS-LBP, and SURF descriptors pass thedistance threshold, but not others, indicate a match.

In some implementations employing descriptor fusion, a single descriptorvector for an identified point can be obtained by combining all or asubset of the different types of descriptors used, e.g., SURF, HoG,EMR-LBP, EMR-CS-LBP, PH-EMR-LBP, and PH-EMR-CS-LBP descriptors, afternormalizing the descriptors to be combined.

In some implementations, employing match metric based fusion, normalizeddistance scores from individual comparisons of different descriptorsbetween enrollment and verification templates can be combined using aweighted average before comparing to a distance threshold to findcorresponding matched pairs of interest points between the twotemplates. In multi-scale matching, the identified template points andtheir descriptors from different scales (e.g., scale 0, scale 1, andscale 2 from an image pyramid) of one template can be matched separatelywith those of the corresponding scales from the other template, orcoordinates of points from lower scales can be up-scaled to scale 0prior to matching.

In general, the points between two templates whose descriptors do notmeet the distance threshold can be discarded from subsequent processing.Thereafter, the non-discarded locations can be used to find the inliersubset of point pairs between enrollment and verification images byfitting one or more affine transformation or similar geometrictransformations as described below. A derivative of the number of inlierpoint pairs, their location correlations, and the required transformscale and rotation can then be used to generate the first stage matchscore. Other match score generation methods, including those taking intoaccount descriptor similarity scores, can also be used.

In some implementations, a random sample consensus (RANSAC) or otheroutlier detection methods can be used to determine the transformationneeded to align candidate points in the verification template withpoints of interest in the first enrollment, where the aforesaid pointsare the point pairs found in preceding descriptor match process. ARANSAC outlier detection method can reject outliers that do not fit ahypothesized geometric transformation between the corresponding pointsin a matched point pair, e.g., in terms of geometries of ocular regionsof interest encoded in enrollment and verification templates viavascular patterns. For example, one or more transformation matrices canbe applied to the points from the verification template in the matchedpairs, generating a set of transformed points that are aligned with thecorresponding points of the enrollment template in terms of theircoordinates, if there is genuine match. The involved transformation canbe derived from the largest consensus (inlier) subset of matched pointpairs based on their descriptor matches. Hypothesis based outlierdetection and image registration methods, such as RANSAC, can be used toidentify one or more affine transformations or similar transformationsthat produce transformed template locations with the most inlier pointpairs. An inlier point pair can include a point in a verificationtemplate that can be aligned to a point in enrollment template using aderived transformation matrix such that the Euclidean or other distancemetric between the aligned points' locations does not exceed a distancethreshold. An inlier point pair can also be a pair of points that yieldsthe closest distance in descriptor space compared to otherenrollment-verification point pairs and successfully survives RANSAC orsimilar outlier detection process. The outlier detection process assumesa geometric or limited elastic transform between inlier point pairlocations from enrollment-verification template comparisons. In general,transformed points from verification template that are inliers arereferred to as aligned points. The transformation of the verificationimage is not performed at stage 1.

A stage 1 match score can be generated based on a function of thecorrelation score of the inlier points across the enrollment andverification templates, plus a function of the number of inliers andrecovered scale and rotation factors from the detected transformationmatrix. In some implantations, other characterizing functions of thetransformation matrix or matrices can be used. In some implementations,similarity metrics other than correlation can be used. For example, thenumber of inlier point pairs, N, can be used to measure the similaritybetween the enrollment and verification templates. A high number ofinlier point pairs, for example, can indicate a higher first stage matchscore than a relatively low number of inlier point pairs. In someimplementations, the correlation score is based on, for example, adistance between registered inlier point locations across the enrollmentand verification templates, the distances between descriptors ofcorresponding matched points from the enrollment and verificationtemplates, a correlation between the locations of the matchedconstellation of points between the enrollment and verification templatepoints, a recovered registration scale and/or rotation between the twotemplates, which may be required for geometric alignment, or acombination of one or more of these and/or other measures. Thecorrelation score can be used alone or in combination with anothermetric to determine the stage 1 match score.

In one example, the match score can be determined by calculating the x,y coordinate correlation between inlier interest points in theverification template and the corresponding points in enrollmenttemplate, and multiplying the correlation coefficient by N, i.e., thenumber of inlier point pairs.

In some implementations, the stage 1 match score is a normalized inlierpairs location' correlation score. In other implementations, the stage 1match score (FMS) can be calculated as:

${FMS} = \frac{\left( \frac{{Cx} + {Cy}}{2} \right)*{\log_{2}(N)}}{\left( {1 + {{\log_{2}\left( {{RS} + 0.001} \right)}}} \right)*\left( {1 + \left( \frac{RA}{0.2} \right)^{2}} \right)}$where C_(x) and C_(y) are the correlation scores between the x and ycoordinates, respectively, of the corresponding enrollment andverification template inliers points. N is the number of these inlierpoint pairs, RA is the recovered angle which represents a change inangle resulting from the transformation of the inlier matchedverification points to the enrollment points, for registration, and RSis the recovered scale which represents a change in scale resulting fromthat transformation. RA and RS can be derived from the affine or similartransformation matrix resulting from RANSAC or similar operation that isused to identify the inlier point pairs.

The first stage match score, in terms of a function of Cx, Cy, N, andthe transformation matrix-derived parameters, can also be directlylearned using labeled dataset of impostor and genuine comparisons totraining a classifier such as artificial neural network or lineardiscriminant analysis. Optionally, a principal component analysis (PCA)preprocessing stage maybe applied before classification. In someimplementations, the local (non-binary) descriptors can be shortenedusing a PCA projection to eliminate post-PCA dimensions that do notcontribute to local descriptor variances. This can improveclassification accuracy while reducing feature dimensionality. Thepercentage of total variance retained for each family of descriptor setcan vary. For instance, in one implementation of PCA projection andshortening, the dimensionality of pattern histograms of extendedmulti-radii local binary pattern features can be reduced to retain about86% of their variance post-PCA shortening. Similarly, SURF basedfeatures can have their dimensionality reduced to retain about 85% oforiginal variance through PCA, and pattern histograms of extendedmulti-radii center symmetric local binary patterns can be shortened toretain about 95% of their variance post PCA projection and shortening.The PCA loadings can be pre-calculated using a training database ofocular templates. Other percentages of variance shortening are possible;they depend on the Sharpening methods and noise levels in an image.

In some implementations, instead of transformation-matrix-derivedparameters RS and RA, another function of the transformation can bedirectly calculated from the its matrix elements. Considering thetransformation matrices derived from genuine and impostor comparisons,it is desired to create a function of corresponding transformationmatrix elements yielding maximum genuine-impostor separation in itsdistribution. One way to achieve this end is to use the transformationmatrices of a labeled dataset of impostor and genuine comparisons totrain a regression function and maximize a measure of classifiability,such as Fisher discriminant ratio.

In some implementations, when multiple image scales are used to generatea multi-scale template of an image region, the point coordinates (butnot the corresponding descriptors) that are not from the original scalecan be multiplied by a scaling factor to project them to the originalscale, combining inlier points (i.e., the corresponding points in theinlier point pairs) from all the scales by projecting them into theoriginal scale. In some implementations, the stage 1 match score can bea weighted sum of several correlation scores from different RANSACtransformations generated from different combinations of interest pointfinders and feature descriptor types. In some implementations, theinlier points from the verification template can be replaced by thealigned points of geometrically transformed template to generate thestage 1 match score described above.

The stage 1 match score can be used, either individually or incombination, with one or more other measures to determine whether averification template is similar enough to an enrollment template, so asto authenticate a user. In some implementations, if the recovered scaleRS is below or above certain values, and/or if the recovered angle RA isabove a certain threshold, a decision not to authenticate the user canbe made using the stage 1 match score, and a stage 2 match score is notcomputed. In some implementations, if such failure to register occurs, adifferent or no registration algorithm can be used to still enable thestage 2 matching described below.

Stage 2 Pattern Matching

The verification image region is transformed (registered) for stage 2matcher using a transformation matrix from the outlier detection process(e.g., RANSAC process) that can align points of the verification imageregion to points of the enrollment image region represented by theenrollment template (step 114).

In some implementations, the transformation includes one or more affineor similar transformations that are applied to the verification imageregion. For example, the verification image region can be translated,scaled, skewed, and/or rotated to generate a transformed image regionwherein points of the transformed image region are in positions similarto the positions of corresponding points in the enrollment image region.

A stage 2 match score can be generated, for example, by matching theoriented local binary patterns of the Gabor filtered enrollment andverification images. In some implementations, the transformedverification image region, after filtering, is used to derive orientedlocal binary patterns (OLBP) image (step 114).

In some implementations, the filtering process includes applying severalconvolutional filters to the transformed verification image region togenerate a filtered verification image region. For example, a set ofcomplex Gabor filters, or a set of complex logarithm of Gabor filters,at various angles can be applied to the transformed verification imageregion (step 112). The parameters of a Gabor filter can be determinedempirically so as to account for variations in the spacing, orientation,and girth of the blood vessels depicted in an image region. The phase ofa complex Gabor filtered image generally reflects the vascular patternsat different angles. The phase of the Gabor-filtered images can varyfrom −π to +π radians. For example, in the phase image filtered by a setof Gabor Kernels (for example, wavelength=6 pixel; spread (standarddeviation) in x=2.5 pixel; spread in (standard deviation) y=2.5 pixel;angles=0⁰, 30⁰, 60⁰, 90⁰, 120⁰, 150⁰) the phase values above 0.25 andbelow −0.25 radians may correspond to vascular structures. Thresholdingthe phase image is not confined to 0.25 or −0.25, and this can bechanged based on application and set of Gabor kernels used.

In some implementations, to threshold the phase image, all values ofphase above 0.25 or below −0.25 are maintained and the remaining valuesare set to zero to derive a thresholded image. This can result in asharper depiction of the vasculature structure that is substantiallyfree of noise in the corresponding phase image. This operation can beperformed for images resulting from applications of several Gaborkernels at different angles. In some implementations, the resultingthresholded images can be added, resulting in a filtered image designedto reveal a fine and crisp vascular structure, such as that depicted inFIG. 6.

In some implementations, in generating stage 1 match score, the imageregions to which the interest point finder (SURF, FAST, HS, and VPD)and/or local image descriptor algorithms (e.g., HoG, SURF, EMR-LBP,PH-EMR-LBP, EMR-CS-LBP and PH-EMR-CS-LBP) are applied as described abovecan be the magnitude of even Gabor filtered image region or magnitude ofsum of all even Gabor filtered image regions at different angles orphase image region or sum of phase image regions at different angles orthresholded phase image regions or sum of all the thresholded phaseimage regions at different angles. In some implementations, a log Gaborkernel can replace Gabor kernel.

In general, the filtered image region can be used to derive an OLBPtemplate (step 114). In some implementations, the filtered image regionis a sum of thresholded phase images at different angles. To generateOLBP image, pixel windows, such as an example 3×3 pixel window depictedin FIG. 7A, are created for each non-border pixel in the filtered imageregion. A 3×3 pixel window generally includes a value for a centerpixel, and values for eight surrounding pixels. Pixel windows of othersizes (e.g., 5×5, 7×7, etc.) can also be used. The values for each pixelcan indicate, for example, an intensity of the corresponding pixel orthe phase information for the pixel. In some implementations, pixelwindows are not created for image border pixels, e.g., pixels at theouter border of the filtered image region, because the border pixels donot have eight surrounding pixels. Binary pixel windows can be createdthat have, for each of the eight surrounding pixels for a center pixel,a binary value that indicates whether the surrounding pixel has a valuegreater or equal than that of the center pixel, or less than the value(i.e., intensity or phase) of the center pixel. For example, the pixelwindow in FIG. 7A is binarized to create a binary pixel window, asdepicted in FIG. 7B, which includes local high values, i.e., “1”s andlogical low values, i.e., “0”s to indicate which surrounding pixelvalues were greater or equal or, alternatively, less than the centerpixel value. For example, with reference to FIG. 7B, a value of “1”indicates that the associated pixel has a pixel intensity value greaterthan or equal to the pixel intensity value of the center pixel, and avalue of “0” indicates that the associated pixel has a pixel intensityvalue less than the pixel intensity value of the center pixel.

For each binary pixel window, a position that corresponds to the centerof the longest string of surrounding “1”s (or, in some implementations,“0”s) is identified (step 114). In the example binary pixel window shownin FIG. 7B, the numbers surrounding the window indicate pixel positions,and the longest string of surrounding “1”s is from position 0 throughposition 3. The center of that string of “1”s is between positions 1 and2, and, in this implementation, the lesser position (i.e., position 1)is identified as the center of the longest string of surrounding “1”s.In some implementations, the greater position (e.g., position 2 in thisexample) can be identified as the center of the longest string of Is.

After identifying the position of the center of the longest string ofsurrounding “1”s, a 4-bit binary Gray code can be generated (step 114).A binary Gray code is string of “1”s and “0”s where each successivevalue differs by only one bit. An example mapping position to Gray codefor the example shown in FIG. 4B is “0001” (as shown in FIG. 7C). Inthis example, pixel position 1 was identified as the center of thelongest string of surrounding “1”s, and position 1 corresponds to Graycode value “0001.” This Gray code is generated for the pixel for whichthe 3×3 pixel window was created (i.e., the pixel in the center of thewindow depicted in FIGS. 7A and 7B). As OLBPs and Gray codes areidentified for every non-border pixel in the filtered enrollment imageregion, each pixel can have a 4-bit Gray code that indicates anorientation of intensity for that pixel.

After generating a Gray code for each non-border pixel of the filteredenrollment image region, four binary layers can be generated for thefiltered enrollment image region. Each binary layer (e.g., in a thirddimension) corresponds to one bit of the 4-bit Gray code. For example,if a pixel at position (10, 10) has a Gray code value of “100,” thebinary value at position (10, 10) of the first binary layer is “1,” thebinary value at position (10, 10) of the second binary layer is “1,” thebinary value at position (10, 10) of the third binary layer is “0,” andthe binary value at position (10, 10) of the fourth binary layer is “0.”To generate stage 2 match score, a similar procedure is applied to thetransformed verification image so as to generate a second Gray codedimage, i.e., the stage 2 verification template, which can be comparedwith the first Gray coded image, i.e., the stage 2 enrollment template.

In some implementations, each binary layer can be tiled forenrollment-verification template comparison (step 114). In one exampleimplementation, the binary layers are tiled into a 4×6 grid of 24 tiles.Tiling can avoid or minimize regions that do not include visiblevasculature and, as such, are not significant for authentication. Tilingcan also minimize registration artifacts. In some implementations,invalid tiles are identified and discarded from further analysis. Forexample, if the area corresponding to a particular tile does not includemuch visible eye vasculature, or includes a large portion of the skin oriris, the tile can be determined to be invalid. This validitydetermination can be made, for example, by comparing a sum of binaryvalues of the area included in a tile to a threshold value, usingeyelash detection algorithms, and/or using glare detection algorithms.In some implementations, a collection of aberration pixels (e.g.detected glare and eyelashes) that are within the white of eye, which inturn is determined by the segmentation process, can be generated.Whether one or more tiles are invalid can be determined based on theratio of the number of aberration pixel counts to the number of thewhite of the eye pixels under the corresponding tiles.

The area of a tile within the white of an eye divided by the total areaof the tile can reflect the extent of sclera within the tile. As such,in one implementation, tiles with less than 80% coverage within thesclera mask can be considered invalid and be therefore dropped. In somecases, portions of a tile area can be occluded by glare or eyelash oreyelid or other artifacts and occlusions, which if severe enough, canresult in the tile being invalidated. The sclera mask is typically abinary image mask that excludes image pixels that do not belong to whiteof the eye. In some implementations, a measure of presence ofvasculature can be used to discard the non-vascular tiles. For example,the area of tile that has the thresholded phase of Gabor values greaterthan zero divided by the total area of the tile can detect the amount ofvisible vasculature.

In various implementations, to determine a stage 2 match score, each bitin each binary layer of the Gray coded stage 2 verification template iscompared to a corresponding bit in the corresponding layer of the Graycoded stage 2 enrollment image (step 114). For example, four binarylayers of a stage 2 verification template can be compared in alayer-by-layer manner against the corresponding four layers of a stage 2enrollment template. In some implementations, a stage 2 match score isbased on a Hamming distance between the corresponding binary layers ofthe Gray coded verification and enrollment stage 2 templates,respectively. In some implementations, a stage 2 match score is based ona correlation between binary layers of the Gray coded stage 2verification and enrollment templates. Sliding window correlation can beused to determine a correlation score for tiles of a binary layer. Inimplementations where binary layers are tiled, only the distance orcorrelation between valid tiles can be used for determining stage 2match score. In some implementations, the Hamming distance can benormalized, e.g., to a value between 0 and 1, for the entire imageregion, or for each layer. For example, the normalized Hamming distancefor each layer can be a number between 0 and 1, where “1” indicates anexact match (no difference) between binary values of a layer of the Graycoded stage 2 verification template and binary values of thecorresponding layer of the stage 2 Gray coded enrollment template, and“0” indicates no matches. In some implementations, the correlation canbe normalized, e.g., to a value between −1 and 1, for the entire imageregion, or for each layer.

The stage 2 match score can be generated, for example, by adding thenormalized Hamming distances calculated for each pair of correspondingbinary layer of the Gray coded stage 2 verification and enrollmenttemplates, resulting in a stage 2 match score between 0 and 4. The stage2 match score can, in some implementations, be further normalized to avalue between 0 and 1. In some implementations, the stage 2 match scorecan be generated based on a normalized Hamming distance between tiles,e.g., by multiplying the number of valid tiles with a mean of normalizedHamming distance calculated across all valid tiles. For example, withfour layers and 10 valid tiles in each layer, a stage 2 match score canbe between 0 and 40, i.e., the sum of normalized Hamming distance foreach of the tiles.

In some implementations, the stage 2 match score can be generated, forexample, by adding the normalized correlation score calculated for eachbinary layer across the Gray coded stage 2 verification and enrollmenttemplates, resulting in a stage 2 match score between −4 and 4. Thestage 2 match score can, in some implementations, be further normalizedto a value between −1 and 1. As another example, the stage 2 match scorecan be generated based on a normalized correlation between tiles, e.g.,by multiplying the number of valid tiles with mean of normalizedcorrelation calculated across all valid tiles. In some implementations,the stage 2 match score generated by correlation score can be comparedto a threshold to determine whether a user providing the verificationimage can be authenticated. For example, if the stage 2 match score isless than 1.0 (on a scale of −4 to 4), the verification attempt can berejected and the user is determined to be unauthorized.

Fusion

In two-stage fusion, the stage 2 match score can be combined with thestage 1 match score, to generate a third (final) match score (step 118).As such, in some implementations, the stage 1 match score and stage 2match score can be multiplied and/or summed to generate the third matchscore. In some implementations, the third match score can be a weightedsum of the stage 1 match score and the stage 2 match score. Weights canbe determined empirically based on historical match records. Forexample, historical instances of authentication failure and success canbe analyzed to determine if the stage 1 match score is more indicativeof an actual match than the stage 2 match score, or if certain matchscore values are more or less indicative of an actual match, and thecorresponding data can be used to train one or more weights for thestage 1 and stage 2 match score. The third match score can be comparedto a threshold score to determine if a verification image matches anenrollment image of an eye. In some implementations, the min or maxfusion rule or a linear discriminant could be used to combine stage 1and stage 2 match scores to generate a third match score.

In one implementation, several stage 1 scores are obtained, each onebased on a different type of image sharpening, a different type ofdescriptor and a different RANSAC run. A weighted summation can be usedto generate a fusion match score based on the various stage 1 matchscores and one stage 2 match score. In one example, the following scoresare obtained by matching stage 1 verification and enrollment templates:

Score1=Stage 1 (Point Finder=FAST and VPD at Scale 0 and Scale 1;Feature Descriptors=EMR-LBP)

Score2=Stage 1 (Point Finder=FAST and VPD at Scale 0 and Scale 1;Feature Descriptors=PH-EMR-LBP)

Score3=Stage 1 (Point Finder=FAST and VPD at Scale 0 and Scale 1;Feature Descriptors=PH-EMR-CS-LBP)

Score4=Stage 1 (Point Finder=FAST and VPD at Scale 0 and Scale 1;Feature Descriptors=SURF)

Score5=Stage 2 score using the transformation matrix corresponding toScore2.

The fusion score can be a weighted sum of all of the above-describedscores, given by:Fusion Score=0.1*Score1+0.2*Score2+0.2*Score3+0.2*Score4+0.1*Score5The weights and combinations in the above example are for oneimplementation. Other combinations of pyramid scales, pyramid types,point finders, and feature descriptors can be employed. In someimplementations, two or more stage 2 match scores can also be includedin the fusion score.

In another implementation, the following scores are obtained by matchingstage 1 verification and enrollment templates:

Score1=Stage 1 (Point Finder=FAST and VPD at Scale 0 and Scale 1;Feature Descriptors=PH-EMR-LBP)

Score2=Stage 1 (Point Finder=FAST and VPD at Scale 0 and Scale 1;Feature Descriptors=PH-EMR-CS LBP)

The fusion score can be a weighted sum of the above-described scores,given by:Fusion Score=0.5*Score1+0.5*Score2

In another implementation, several stage 1 scores are obtained byapplying several different RANSAC procedures and one stage 2 score isobtained. These scores can be combined using a weighted summation togenerate the fusion match score. In one example, the following scoresare obtained by matching verification and enrollment templates:

-   -   Score1=Stage 1 (Point Finder=FAST and VPD at Scale 0; Feature    -   Descriptors=EMR-LBP, PH-EMR-LBP, PH-EMR-CS-LBP, SURF, HoG;        RANSAC is run on points out of match metric based fusion)    -   Score2=Stage 1 (Point Finder=FAST and VPD at Scale 1; Feature    -   Descriptors=EMR-LBP, PH-EMR-LBP, PH-EMR-CS-LBP, SURF, HoG;        RANSAC is run on points out of match metric based fusion)    -   Score3=Stage 2 score, using the transformation matrix derived        from Score1.        The fusion score can be a weighted sum of all of the scores,        given by:        Fusion Score=0.4*Score1+0.3*Score2+0.3*Score3        It should be understood that the weights and combinations used        in the examples above are illustrative only, and that other        weights, number and types of descriptors, and RANSAC runs        (inlier detection procedures, in general) can be employed. Other        combinations of point finders, feature descriptors, pyramid        scales, and pyramid types for match metric based fusion can be        employed as well.

In another implementation, several stage 1 scores are obtained byapplying several different RANSAC procedures on differently sharpenedimages. These scores can be combined using a weighted summation togenerate the fusion match score. In one example, the following scoresare obtained by matching verification and enrollment templates:

-   -   Score1=Stage 1 (Sharpening=LoG based; Point Finder=FAST and VPD        at Scale 0; Feature Descriptors=EMR-LBP, PH-EMR-LBP,        PH-EMR-CS-LBP, SURF, HoG; RANSAC is run on points out of match        metric based fusion)    -   Score2=Stage 1 (Sharpening=LoG based; Point Finder=FAST and VPD        at Scale 1; Feature Descriptors=EMR-LBP, PH-EMR-LBP,        PH-EMR-CS-LBP, SURF, HoG; RANSAC is run on points out of match        metric based fusion)    -   Score3=Stage 1 (Sharpening=Gabor based; Point Finder=FAST and        VPD at Scale 0; Feature Descriptors=EMR-LBP, PH-EMR-LBP,        PH-EMR-CS-LBP, SURF, HoG; RANSAC is run on points out of match        metric based fusion)    -   Score4=Stage 1 (Sharpening=Gabor based; Point Finder=FAST and        VPD at Scale 1; Feature Descriptors=EMR-LBP, PH-EMR-LBP,        PH-EMR-CS-LBP, SURF, HoG; RANSAC is run on points out of match        metric based fusion)        The fusion score can be a weighted sum of all of the scores,        given by:        Fusion Score=0.3*Score1+0.2*Score2+0.3*Score3+0.2*Score4        It should be understood that the weights and combinations used        in the examples above are illustrative only, and that other        weights, number and types of descriptors, and RANSAC runs        (inlier detection procedures, in general) can be employed. Other        combinations of point finders, feature descriptors, pyramid        scales, and pyramid types for match metric based fusion can be        employed as well.

In some implementations, the fusion score is obtained using a singleenrollment and a single verification template. In some implementations,a final match score can be generated by comparing one or moreverification templates with one or more enrollment templates. Forexample, if there are two verification templates and two enrollmenttemplates, four fusion scores can be generated. In some implementations,the final match score can be generated using a max rule or a sum rule.In other implementations, the match score of the highest N′ inlierspoints (identified via several outlier detection runs) and/or the bestquality score is selected as the final match score. In someimplementations, match scores are generated serially until the matchscore reaches certain threshold or until all or a predefined number ofselected comparisons are performed. In some implementations, the stage 1match score can be used in combination with a stage 2 match score togenerate a third match score for determining a degree of similaritybetween a verification template and an enrollment template. In someimplementations, by way of match score fusion, the stage 1 match scorescan be used to generate a third match score for determining a degree ofsimilarity between a verification template and an enrollment template.In some implementations, by way of fusion, the stage 2 match scores canbe used to generate a third match score for determining a degree ofsimilarity between a verification template and an enrollment template.

Rolling Template Update and Intelligent Template Recall forMulti-Template Matching

FIG. 8 illustrates an example process of updating enrollment templateson a rolling basis. In order to efficiently manage several enrollmenttemplates per each ocular ROI of a user, the enrollment templates storedin one or more template banks (e.g., the database 120 shown in FIGS. 1Aand 1B) can be refined and updated. The number of enrollment templatesto be stored can be optionally reduced at the time of enrollment. Tothis end, in one implementation, the initial N enrollment templates fromeach ROI are matched in a pair-wise manner against each other, and onlythose templates having the highest overall cross match scores are stored(step 802). For example, given the initial N enrollment templates, theN(N−1)/2 possible pairs are matched assuming a symmetric distancemetric. Then, the template with the lowest median match score isexcluded. The median match score can be a median of the N−1 match scoresfor each enrollment template, each one corresponding to a comparison ofthe template being tested for inclusion in an enrollment bank with theremaining N−1 enrollment templates. This procedure can be repeated toomit from the template bank or banks one or more additional enrollmenttemplates.

In some implementations, a quality score is generated for all enrollmenttemplates and the verification template. The verification and enrollmenttemplates include sparse vascular points that are based on binaryvascular maps (BVMs). In some implementations, the quality scorecorresponding to a template includes the ratio of true pixels (i.e.,pixels designated a logical high value) in a thinned BVM associated withthe template to the number of true pixels in the scleral mask used ingenerating that BVM. In another method, quality score of an enrollmentand/or verification template can be calculated as a ratio of true pixelsin a BVM associated with the template to the number of true pixels inthe scleral mask. Quality score can provide a measure of amount ofvisible vasculature present in an ocular image in order to assess theeligibility thereof for further processing in a matching process.Enrollment images having a quality score below a certain threshold arenot processed for inclusion in enrollment bank.

In some embodiments, the template bank or banks can be additionally orin the alternative, updated at the time of verification by addingqualified verification template(s) as additional enrollment template(s)or by replacing previously stored enrollment templates of lower qualitywith relatively better quality verification templates. To qualify, anincoming verification template has to match well with one or moreenrollment templates, and optionally pass the earlier mentioned qualitymeasure for that verification template to be added to the template bank(step 804). If the template bank cannot store the additional template,e.g., due to memory capacity constraint, number of enrollment templatesconstraint, etc., the least desirable previously stored enrollmenttemplate can be excluded, e.g., using the above process (step 806). Insome embodiments, the lower quality enrollment template is preserved,nevertheless, if a larger enrollment bank is required, e.g., due to lackof sufficiently diverse templates from initial registration process,where diversity is defined as a measure of externally induced variationsin templates while the genuine user scans his or her eyes under varyingconditions.

In some embodiments, an exposure profile of each template in amulti-enrollment template system is also stored along with the templateas a measure of template diversity (e.g., in step 108 shown in FIG. 1A).The exposure profile of each image corresponding to a template to besaved can be computed over a pre-registered ROI using a number ofmethods. Example methods include using the camera's intrinsic exposuremetering variables (e.g. those found in the image's EXIF file), usingthe median, (mean, standard deviation) pair, and/or a histogram of the Ycomponent in the YUV presentation of the image, or just the green layer,for exposure profile. In the latter case, to find the exposuresimilarity score between two captures, a histogram distance measure,such as Kolmogorov-Smirnov, normalized absolute difference, histogramintersection, mean squared error, Chi-squared based distance,Kullback-Leibler divergence, Bhattacharyya distance, or correlationcoefficient, can be used. Sensitivity towards asymmetric lighting andspatial distribution of exposure can be increased by splitting the imageinto two or four segments (or other spatial arrangements, which can beoverlapping), and by concatenating the above-described exposure measurescalculated per each segment, before measuring exposure profilesimilarities.

The procedures described above are relatively straightforwardstatistical image similarity measures that can mostly reveal exposureand/or lighting induced differences between templates. To improve theaccuracy of the similarity measure, the images can be pre-registered andcropped to the ROI of choice, e.g., a bounding box of whole eye orsegmented sclera of specific gaze direction. In one implementation,pre-registration can be performed by finding a scale from inter-oculardistance, and translation from a combination of center of iris and/orsclera and/or eye. A rotation angle can be determined from a lineconnecting the former two points, and pre-registering can beaccomplished using a similarity geometric transformation based on thescale, rotation angle, etc. During verification, the matching of anincoming verification template can start with the enrollment templatesthat have the most similar exposure profiles, which can reduce the matchtime in the first-best multi-comparison, i.e. by exiting with a matchdecision as soon as a match threshold is reached.

In some implementations, for exposure-aware rolling template update, thecalculated match metric is modulated allowing for exposure diversity byproviding a limited advantage to templates having different lighting andexposure conditions that those of the enrollment template (step 118,depicted in FIG. 1B). Such differences can put these verificationtemplates at a disadvantage otherwise because their match scores arelower at least in part due to the exposure difference. In general, it isbeneficial not to allow a significant exposure difference between theverification and enrollment templates to overcome the modulated metric,thereby erroneously allowing a close impostor template to be added tothe enrollment template bank. Therefore, for templates T₁ and T₂,according to one linear implementation:enhanced_match_metric(T ₁ ,T ₂)=a*match_score(T ₁ ,T₂)+(1−a)*(min(exposure_difference(T ₁ ,T ₂),influnce_threshold)

The influnce_threshold can ensure that the exposure difference leveragedoes not exceed a certain level. A suitable influence_threshold can bedetermined according to parameter ‘a’ for best performance, over alabeled training dataset. According to the above enhanced match metric,a rather weak match but with a significant exposure difference betweenimages that produced templates T₁ and T₂ can be a strong match withoutsuch exposure difference and, as such, the incoming verificationtemplate can be leveraged in order to be selected in a rolling templateupdate. A measure of vascularity/image quality can be added to thisformula to further ensure that templates from low-quality images (i.e.,images lacking adequate well defined vascular structure due to blur,reflections and occlusions, etc.) are also not selected as enrollmenttemplates.

In one implementation of the histogram-based exposure similarity metric,an ocular image is cropped to the bounding box of a scleral mask and ispadded with a certain number of pixels (e.g., about 50 pixels), orotherwise the ocular image centered with respect to the eye. Then, a 64bin histogram of the green layer of each quadrant of a first and secondimage is calculated. Other histograms such as 32 bin or 128 binhistograms can also be used. These histograms are stored alongside theircorresponding templates as exposure profiles or exposure templates.Next, histogram distance or similarity metrics between histograms ofeach corresponding quadrants between the pair of templates iscalculated. Specifically, the following metrics are calculated:normalized absolute value of histogram differences (ABSNdist), histogramintersection similarity (INTRsim), correlation coefficient similarity ofhistograms (CORRsim), and their Bhattacharyya distance (BHATdist). Inother implementations, fewer and/or other metrics can be used. Finally,these metrics are combined into a single similarity metric as:

similarity = −1 * ABSNdist 2 − 1 * ABSNdist 1 − 1 * ABSNdist 3 − 1 * ABSNdist 4 − 1 * BHATdist 2 − 1 * BHATdist 1 − 1 * BHATdist 3 − 1 * BHATdist 4 + INTRsim 2 + INTRsim 1 + INTRsim 3 + INTRsim 4 + CORRsim 2 + CORRsim 1 + CORR sim 3 + CORRsim 4;The higher the similarity metric, the more similar the two templatesare. Dissimilarity or exposure difference can be a negated value of thesimilarity metric.

The systems and techniques described here can be implemented in acomputing system that includes a back end component (e.g., as a dataserver), or that includes a middleware component (e.g., an applicationserver), or that includes a front end component (e.g., a client computerhaving a graphical user interface or a Web browser through which a usercan interact with an implementation of the systems and techniquesdescribed here), or any combination of such back end, middleware, orfront end components. The components of the system can be interconnectedby any form or medium of digital data communication (e.g., acommunication network). Examples of communication networks include alocal area network (“LAN”), a wide area network (“WAN”), and theInternet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. Anumber of embodiments have been described. Nevertheless, it will beunderstood that various modifications may be made without departing fromthe spirit and scope of the invention.

Embodiments of the subject matter and the operations described in thisspecification can be implemented in digital electronic circuitry, or incomputer software, firmware, or hardware, including the structuresdisclosed in this specification and their structural equivalents, or incombinations of one or more of them. Embodiments of the subject matterdescribed in this specification can be implemented as one or morecomputer programs, i.e., one or more modules of computer programinstructions, encoded on computer storage medium for execution by, or tocontrol the operation of, data processing apparatus. Alternatively or inaddition, the program instructions can be encoded on anartificially-generated propagated signal, e.g., a machine-generatedelectrical, optical, or electromagnetic signal, that is generated toencode information for transmission to suitable receiver apparatus forexecution by a data processing apparatus. A computer storage medium canbe, or be included in, a computer-readable storage device, acomputer-readable storage substrate, a random or serial access memoryarray or device, or a combination of one or more of them. Moreover,while a computer storage medium is not a propagated signal, a computerstorage medium can be a source or destination of computer programinstructions encoded in an artificially-generated propagated signal. Thecomputer storage medium can also be, or be included in, one or moreseparate physical components or media (e.g., multiple CDs, disks, orother storage devices).

The operations described in this specification can be implemented asoperations performed by a data processing apparatus on data stored onone or more computer-readable storage devices or received from othersources.

The term “data processing apparatus” encompasses all kinds of apparatus,devices, and machines for processing data, including by way of example aprogrammable processor, a computer, a system on a chip, or multipleones, or combinations, of the foregoing The apparatus can includespecial purpose logic circuitry, e.g., an FPGA (field programmable gatearray) or an ASIC (application-specific integrated circuit). Theapparatus can also include, in addition to hardware, code that createsan execution environment for the computer program in question, e.g.,code that constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, a cross-platform runtimeenvironment, a virtual machine, or a combination of one or more of them.The apparatus and execution environment can realize various differentcomputing model infrastructures, such as web services, distributedcomputing and grid computing infrastructures.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, object, orother unit suitable for use in a computing environment. A computerprogram may, but need not, correspond to a file in a file system. Aprogram can be stored in a portion of a file that holds other programsor data (e.g., one or more scripts stored in a markup languageresource), in a single file dedicated to the program in question, or inmultiple coordinated files (e.g., files that store one or more modules,sub-programs, or portions of code). A computer program can be deployedto be executed on one computer or on multiple computers that are locatedat one site or distributed across multiple sites and interconnected by acommunication network.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back-end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front-end component, e.g., aclient computer having a graphical user interface or a Web browserthrough which a user can interact with an implementation of the subjectmatter described in this specification, or any combination of one ormore such back-end, middleware, or front-end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include a local area network (“LAN”) and a widearea network (“WAN”), an inter-network (e.g., the Internet), andpeer-to-peer networks (e.g., ad hoc peer-to-peer networks).

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someembodiments, a server transmits data (e.g., an HTML page) to a clientdevice (e.g., for purposes of displaying data to and receiving userinput from a user interacting with the client device). Data generated atthe client device (e.g., a result of the user interaction) can bereceived from the client device at the server.

A system of one or more computers can be configured to performparticular operations or actions by virtue of having software, firmware,hardware, or a combination of them installed on the system that inoperation causes or cause the system to perform the actions. One or morecomputer programs can be configured to perform particular operations oractions by virtue of including instructions that, when executed by dataprocessing apparatus, cause the apparatus to perform the actions.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinventions or of what may be claimed, but rather as descriptions offeatures specific to particular embodiments of particular inventions.Certain features that are described in this specification in the contextof separate embodiments can also be implemented in combination in asingle embodiment. Conversely, various features that are described inthe context of a single embodiment can also be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

Thus, particular embodiments of the subject matter have been described.Other embodiments are within the scope of the following claims. In somecases, the actions recited in the claims can be performed in a differentorder and still achieve desirable results. In addition, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In certain implementations, multitasking and parallelprocessing may be advantageous.

What is claimed is:
 1. A computer-implemented method comprising:receiving a verification template comprising a plurality of points ofinterest, each point of interest being associated with a plurality ofdifferent respective descriptors that each describe one or morelocalities surrounding the corresponding point of interest; computing amatch score for the verification template by comparing the verificationtemplate with one or more enrollment templates in a plurality ofenrollment templates; and adding the verification template to theplurality of enrollment templates based on both of (i) the match scorematching or exceeding an enrollment threshold and (ii) respectivequality scores associated with the verification and enrollmenttemplates, wherein each of the verification template and the enrollmenttemplates comprises vascular points based on a binary vascular map(BVM), and wherein the quality score associated with a particularverification or enrollment template comprises a ratio of a number oftrue pixels in the BVM corresponding to that template and a number oftrue pixels in a scleral mask used in determining the BVM.
 2. The methodof claim 1, further comprising adjusting the match score according to(i) an exposure difference between a verification exposure template andan enrollment exposure template associated with one of the enrollmenttemplates and/or (ii) an influence threshold.
 3. The method of claim 2,further comprising computing the exposure difference and/or an exposuresimilarity, based on at least one histogram measure.
 4. The method ofclaim 1, further comprising generating a verification exposure templateand/or an enrollment exposure template.
 5. The method of claim 4,wherein generating the verification exposure template and/or theenrollment exposure template comprises: partitioning a region ofinterest (ROI) of an image corresponding to the verification templateinto at least two sections; and generating for each section a histogramof intensities representing, for each intensity in the histogram, anumber of pixels in the section of the ROI having substantially thatintensity.
 6. The method of claim 5, further comprising selecting theROI by cropping an image of an eye according to a boundary of a scleralmask.
 7. The method of claim 5, further comprising selecting the ROI bycropping an image of an eye according to a boundary from one eye cornerto another eye corner and from one eye lid to another eye lid.
 8. Themethod of claim 4, wherein generating the verification exposure templateand/or the enrollment exposure template comprises generating a pluralityof exposure measures.
 9. A system comprising: at least one processor;and at least one memory for storing computer-executable instructionsthat, when executed by the at least one processor, program the at leastone processor to perform the operations of: receiving a verificationtemplate comprising a plurality of points of interest, each point ofinterest being associated with a plurality of different respectivedescriptors that each describe one or more localities surrounding thecorresponding point of interest; computing a match score for theverification template by comparing the verification template with one ormore enrollment templates in a plurality of enrollment templates; andadding the verification template to the plurality of enrollmenttemplates based on both of (i) the match score matching or exceeding anenrollment threshold and (ii) respective quality scores associated withthe verification and enrollment templates, wherein each of theverification template and the enrollment templates comprises vascularpoints based on a binary vascular map (BVM), and wherein the qualityscore associated with a particular verification or enrollment templatecomprises a ratio of a number of true pixels in the BVM corresponding tothat template and a number of true pixels in a scleral mask used indetermining the BVM.
 10. The system of claim 9, wherein the operationsfurther comprise adjusting the match score according to (i) an exposuredifference between a verification exposure template and an enrollmentexposure template associated with one of the enrollment templates and/or(ii) an influence threshold.
 11. The system of claim 10, wherein theoperations further comprise computing the exposure difference and/or anexposure similarity, based on at least one histogram measure.
 12. Thesystem of claim 9, wherein the operations further comprise generating averification exposure template and/or an enrollment exposure template.13. The system of claim 12, wherein generating the verification exposuretemplate and/or the enrollment exposure template comprises: partitioninga region of interest (ROI) of an image corresponding to the verificationtemplate into at least two sections; and generating for each section ahistogram of intensities representing, for each intensity in thehistogram, a number of pixels in the section of the ROI havingsubstantially that intensity.
 14. The system of claim 13, wherein theoperations further comprise selecting the ROI by cropping an image of aneye according to a boundary of a scleral mask.
 15. The system of claim13, wherein the operations further comprise selecting the ROI bycropping an image of an eye according to a boundary from one eye cornerto another eye corner and from one eye lid to another eye lid.
 16. Thesystem of claim 12, wherein generating the verification exposuretemplate and/or the enrollment exposure template comprises generating aplurality of exposure measures.
 17. A computer-implemented methodcomprising: receiving a verification template comprising a plurality ofpoints of interest, each point of interest being associated with aplurality of different respective descriptors that each describe one ormore localities surrounding the corresponding point of interest;computing a match score for the verification template by comparing theverification template with one or more enrollment templates in aplurality of enrollment templates; and adding the verification templateto the plurality of enrollment templates based on respective qualityscores associated with the verification and enrollment templates,wherein each of the verification template and the enrollment templatescomprises vascular points based on a binary vascular map (BVM), whereinthe quality score associated with a particular verification orenrollment template comprises a ratio of a number of true pixels in theBVM corresponding to that template and a number of true pixels in ascleral mask used in determining the BVM.
 18. A computer-implementedmethod comprising: receiving a verification template comprising aplurality of points of interest, each point of interest being associatedwith a plurality of different respective descriptors that each describeone or more localities surrounding the corresponding point of interest;computing a match score for the verification template by comparing theverification template with one or more enrollment templates in aplurality of enrollment templates; adding the verification template tothe plurality of enrollment templates based on the match score matchingor exceeding an enrollment threshold, wherein each of the verificationtemplate and the enrollment templates comprises vascular points based ona binary vascular map (BVM); and adjusting the match score according to(i) an exposure difference between a verification exposure template andan enrollment exposure template associated with one of the enrollmenttemplates and/or (ii) an influence threshold.
 19. A computer-implementedmethod comprising: receiving a verification template comprising aplurality of points of interest, each point of interest being associatedwith a plurality of different respective descriptors that each describeone or more localities surrounding the corresponding point of interest;computing a match score for the verification template by comparing theverification template with one or more enrollment templates in aplurality of enrollment templates; adding the verification template tothe plurality of enrollment templates based on the match score matchingor exceeding an enrollment threshold, wherein each of the verificationtemplate and the enrollment templates comprises vascular points based ona binary vascular map (BVM); and generating a verification exposuretemplate and/or an enrollment exposure template.